What is meant by an UTF8 codepoint? I thought that by the time you're in UTF-8 you no longer deal in codepoints, just in bytes?

The Unicode consortium does specify that lengths can be counted in one of 4 ways. From the FAQ using the string aनि亜𐂃 (U+0061, U+0928, U+093F, U+4E9C, U+10083)

• bytes: UTF-8: 14, UTF-16: 12, UTF-32: 20
• code units (position in a string with the given size): UTF-8: 14, UTF-16: 6, UTF-32: 5
• code points (how many unicode code points are in there): 5 for all, since there is a combining mark
• grapheme clusters: 4 for all of them, since this is a logical grouping for the length based on 'characters' as we people usually think of them, rather than the implementation of them.

The two latter are not encoding-specific. What is meant exactly by UTF-8 code points? Just code points, or it's a bad term for code units (and hence bytes)?

Hey @ferd, I meant literally code points as in your definition:

code points (how many unicode code points are in there): 5 for all, since there is a combining mark

It is also worth mentioning the implementation today (prior to this patch) also checks code points. This can be checked in the source code or by trying a quick example:

3> binary_to_atom(binary:copy(<<"é"/utf8>>, 255), utf8).
ééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé
4> byte_size(binary:copy(<<"é"/utf8>>, 255)).
510
5> binary_to_atom(binary:copy(<<"é"/utf8>>, 256), utf8).
** exception error: a system limit has been reached
     in function  binary_to_atom/2
        called as binary_to_atom(<<"éééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééééé"/utf8...>>,
                                 utf8)

So this pull request is currently backwards incompatible (hence the pending tasks). The question is mostly how to implement the new binary_to_atom efficiently.

Patch has passed first testings and has been assigned to be reviewed

I am a script, I am not human

We will not have time to do a thorough review until OTP 19.0 has been released, but here is some quick feedback:

The test case lc_SUITE:effect/1 fails.

binary_to_atom(): Solution i, that is keep the existing error semantics and implement it efficiently.

Not modifying list_to_atom/1 is an interesting idea that we have not considered.

@bjorng Can we name the chunk 'JOSÉ'? PLS.

We will take a closer at this PR request when you have fixed the failed test case mentioned above and rebased it on the latest master.

It is also worth mentioning the implementation today (prior to this patch) also checks code points. This can be checked in the source code or by trying a quick example:

I think you interpreted that incorrectly. Currently only UTF-8 encoded atoms that can be converted to Latin-1 are supported, so it just counts the number of Latin-1 code points, which equates the number of Latin-1 code units, which equates the number of bytes in the result.

@bjorng I will allocate time in the next weeks to work on all of my pending pull requests. :D

While talking to @nox, we came up with another solution for the second bullet above. @nox commented that it would be much simpler to count bytes instead of counting codepoints. However, if we count bytes and keep the limit of 255 bytes, it would be backwards incompatible because an atom made only of the letter é in latin1 will now take double the space in UTF-8, possibly triggering the system limit.

However, if we double the maximum atom size, we could keep the conceptually simpler and faster code that only counts bytes and remain backwards compatible.

From the Elixir perspective, Increasing the atom limit would be a positive change, since function names must be atoms, sometimes we may reach this limit when building functions dynamically.

@bjorng thoughts on this suggestion?

I think that the BIFs that create atoms should count code points, and let the limit be 255 code points. When converting from a list, the number of code points is the number of elements in the list (assuming that we don't try to combine code points that can be combined).

The chunk in the BEAM file can give the size of each atom in bytes, and the loader can trust the compiler not to create overlong atoms. (There could be a sanity check to abort the loading if any atom is longer than 1024 bytes.)

Rationale for counting code points: We don't want to handle support request from Klingon users that complain that they can only put 85 Klingon characters into their atoms, while speakers of most west-European languages can have 255 characters in their atoms.

What about following what ROK proposed in EEP20?

2 bytes for byte length, 2 bytes for code point count.

@bjorng This is ready for review. I have rebased, fixed all feedback and features.

I have ran emulator, compiler and stdlib test suites with the following results:

• emulator: One failure regarding registered sends across nodes (I inspected the code and it seems to not related be to this patch)
• compiler: No failures
• stdlib: One failure in the shell suite which seems unrelated

I have also added a test that compiles a module from the Erlang Abstract Format with utf8 atoms, to ensure the whole compilation chain works, and also a test for the r19 option.

Patch has passed first testings and has been assigned to be reviewed

I am a script, I am not human

Patch has passed first testings and has been assigned to be reviewed

I am a script, I am not human

Three test cases fail in compile_SUITE: core, core_roundtrip, and asm.

I updated the primary bootstrap and compiled from a clean repository.

I updated the primary bootstrap and compiled from a clean repository.

How do I update the primary bootstrap? I remember doing a clean build but I did not touch the bootstrap files. Thanks!

I found and fixed another minor bug. Please squash and rebase on latest master.

Done and done!

Btw the reducing memory during compilation I had to rebase on may help Elixir too. 👍

Can you please rebase again?

We will test the branch in our daily builds again to see that there are no remaining failed test cases.

@bjorng done!

This branch no longer seems to cause any problems in our daily builds. We are thinking about merging it soon. More work is probably needed in some applications, for example to ensure that "~ts" is used as format specifier when an atom is displayed, but we can do that in separate branches.

However, I think that the documentation should be updated in this branch to reflect the changes, at least the BIFs for the atom BIFs. Do you think you could update the documentation?

I will push the docs soon. :) Two questions:

1. Regarding ~ts, I understand why it is needed for binaries and char lists, but it shouldn't be necessary for atoms assuming that atoms do have the encoding they are written in? So I suggest making ~s work regardless of the atom encoding. Or do you see reasons where this would be a bad idea™?

2. Should we explicitly document module names can now be UTF-8? If so, I would like to add a test to compile_SUITE where we compile Erlang Forms module with a UTF-8 atom as name.

I have pushed a separate commit with docs. Once review is done, I can squash it all together.

1. Making ~s work for any atom is an interesting idea. I will discuss it with the team.

2. We have not reached a final decision yet. Our current thinking is that we will allow Unicode characters in module names, but we will not recommend it because of potential portability problems having to do with the file names (Linux does not enforce any particular encoding for file names, and their could be other potential pitfalls when moving files from one operating system to another). The reason that we will probably allow it is because those potential problem already exists for module names with Latin1 characters.

Our current thinking is that we will allow Unicode characters in module names, but we will not recommend it because of potential portability problems having to do with the file names

I agree. Let me know what are your decisions regarding 1 and 2 and I will gladly add tests and improve the docs.

There may some time before we can reach a decision. We will merge this branch and do further corrections/improvement in separate pull requests.

Please squash the commits. When you have done it, we will run the branch once more in our daily builds and then merge it.

Squashed and pushed.

do further corrections/improvement in separate pull requests.

Like renaming the chunk to 'JOSÉ', right?

🎉

Support for decompiling AtU8 chunks has been added to IntelliJ Elixir in KronicDeth/intellij-elixir#777, which will be released in version 6.0.0.